Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sphinx-autoapi to generate API documentation #3646

Open
wants to merge 55 commits into
base: main
Choose a base branch
from

Conversation

valeriupredoi
Copy link
Contributor

@valeriupredoi valeriupredoi commented Jun 6, 2024

Description


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Jun 11, 2024

@bouweandela have you actually run the API docs builds with sphinx-autoapi? I have just now, locally, and it's incredibly slow and spits out a gazillion formatting errors a la:

/home/valeriu/ESMValTool/doc/sphinx/source/autoapi/single_model_diagnostics/index.rst:179: ERROR: Unexpected indentation.
/home/valeriu/ESMValTool/doc/sphinx/source/autoapi/utils/index.rst:130: ERROR: Unexpected indentation.
/home/valeriu/ESMValTool/doc/sphinx/source/autoapi/utils/index.rst:132: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/valeriu/ESMValTool/doc/sphinx/source/autoapi/ww09_esmvaltool/index.rst:41: ERROR: Unexpected indentation.

@valeriupredoi
Copy link
Contributor Author

also any clue how to tell ReadTheDocs we don't want a local build of the package? It complains it's missing the pkg metadata since the darn thing was not pip installed, but we don't want that, that's exactly what we're trying to avoid with autoapi https://readthedocs.org/projects/esmvaltool/builds/24658734/

@bouweandela
Copy link
Member

We probably need a different way of getting the version than importing the whole package (which fails because it is not installed):

from esmvaltool import __version__

@bouweandela
Copy link
Member

have you actually run the API docs builds with sphinx-autoapi?

No

I have just now

🍻

it's incredibly slow

How slow is incredibly slow? The current build on readthedocs takes 15 minutes.

it spits out a gazillion formatting errors

Are those for modules that are part of the public API?

@valeriupredoi
Copy link
Contributor Author

let's see how slow it is on RtD - when it runs, it took me some 10min and I killed it; indeed, all manners of modules that should have private API, think that needs configuring not to go through those, need to see their configuration settings. Ah cheers for the __version__ pointer, have completely overlooked that! 🍺

@valeriupredoi
Copy link
Contributor Author

well good news! It ran and it ran in 6min 🥳 The warnings are annoying though - it's looking for esmvaltool modules in diags - we need to configure it so it doesn't do that; incidentally, RtD's new dashboard is looking nice but it adds a million empty lines to an already very stuffy output https://app.readthedocs.org/projects/esmvaltool/builds/24659446/

@valeriupredoi
Copy link
Contributor Author

OK 396 warnings from intial 470 - I am now excluding diag_scripts - otherwise we get all these sort of warnings:

Argument:
--------
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/ens_eof_kmeans/index.rst:30: ERROR: Unexpected indentation.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/ens_eof_kmeans/index.rst:31: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/autoassess/_plot_mo_metrics/index.rst:187: ERROR: Unexpected indentation.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/autoassess/_plot_mo_metrics/index.rst:188: WARNING: Block quote ends without a blank line; unexpected unindent.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/autoassess/autoassess_area_base/index.rst:151: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/autoassess/autoassess_area_base/index.rst:151: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/autoassess/loaddata/index.rst:167: ERROR: Unexpected indentation.
/home/docs/checkouts/readthedocs.org/user_builds/esmvaltool/checkouts/3646/doc/sphinx/source/autoapi/esmvaltool/diag_scripts/emergent_constraints/lif1f2/index.rst:3: CRITICAL: Missing matching underline for section title overline.

]

# Autoapi configuration
autoapi_dirs = ['../../../esmvaltool']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible change this list so only these are documented? https://github.com/ESMValGroup/ESMValTool/tree/main/doc/sphinx/source/api

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeh lemme add all those to the list that way then

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it works nicely! Less than two minutes to run, but it throws 100+ warnings, see https://readthedocs.org/projects/esmvaltool/builds/24660091/ - I think I know what's going on in here - there are two types of warnings:

  • a lot coming from autodoc which I should turn it off completely
  • another lot coming from autoapi docs dir sitting inside our source so it's kicking its own butt since the autoapi docs are not included in any docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey Bouwe, any ideas how to fix these warnings? https://readthedocs.org/projects/esmvaltool/builds/24661254/ - I've done everything I could think of, and a tap dance on top of it too - autodoc keeps wanting to import modules, even though I added ESMValTool and esmvaltool to the Python path (that's just not how things should be done, but that's what the Sphinx folks say we should do), and the blithering autoapi keeps creating that index.rst even if I told it no with autoapi_generate_api_docs = False (see https://sphinx-autoapi.readthedocs.io/en/latest/how_to.html). I am stuck, and am gonna leave it for now 😁

Copy link
Member

@bouweandela bouweandela Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I fixed the build in f07c760, but it is not pretty. Somehow I cannot get autoapi to respect what is in __all__ (see here). At least in part that seems to be because it also searches for executable scripts and documents those and those appear to be scattered throughout the code base. Maybe if we clean that up things will work better?

We would still need to move the documentation from the now deleted doc/sphinx/source/api directory to the Python modules and do some further polishing before this is ready.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a serious bit of progress - from 126 to 7 warnings, bud 🍺 But what do you mean moving the docs to the Python modules, as in create the docs in the same place where the source code lives? Lemme see about restricting Autoapi to not look for executables, that should have a configuration solution

Copy link
Member

@bouweandela bouweandela Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not executable scripts after all, it is missing __init__.py files that generate all the extra packages. Let me try to add some.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahaa! That's me completely unawares - it's great you figured that out otherwise it'd have taken me until 2027 to 😁 So I figured out the tests issue and now I think I just need to add Katja's testkw to the ignored list too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But what do you mean moving the docs to the Python modules, as in create the docs in the same place where the source code lives?

Copy the text from the deleted .rst files to the module docstrings

@valeriupredoi valeriupredoi marked this pull request as ready for review June 20, 2024 13:45
@valeriupredoi valeriupredoi requested a review from a team as a code owner June 20, 2024 13:45
@valeriupredoi valeriupredoi removed the request for review from a team June 20, 2024 13:47
@valeriupredoi valeriupredoi added this to the v2.12.0 milestone Jun 20, 2024
@bouweandela
Copy link
Member

This is not quite ready for review yet: the text from the removed .rst files still needs to be moved to the docstrings of the relevant python modules.

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Jun 21, 2024

This is not quite ready for review yet: the text from the removed .rst files still needs to be moved to the docstrings of the relevant python modules.

well, you are the only reviewer, and major contributor, so this is ready only when you say so, bud - plop a changes needed blocker and we is good 🍺

@valeriupredoi
Copy link
Contributor Author

@bouweandela am gonna fix the myriad of conflicts now, then maybe you have a second or two give it a bit more polish, and merge? It don't have to be perfect 😁

@valeriupredoi
Copy link
Contributor Author

@bouweandela you reckon we can get this in? Or, you'd still want to prettify the docs layouts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider using sphinx-autoapi
2 participants